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I, DMITRY I. BELOV, hereby declare the following: 

1 . I am the sole inventor of the subject matter claimed in the above-referenced patent 
application. 

2. I am a co-author of a publication cited in an Information Disclosure Statement filed 
September 9, 2004, entitled Belov, D. and Armstrong, R.D., A Stochastic Search for Test 
Assembly, Item Pool Analysis, and Design, ("Stochastic Search"). 

3 . Stochastic Search describes my own work. 

4. The contributions made by Ronald D. Armstrong to the Stochastic Search are identified 
below and are not claimed in the above-referenced patent application: 

a) The use of a "maximum set packing" formulation, on page 10, in the paragraph before 
Algorithm 2 of the cited document and as set forth below in the Stochastic Search. The use of 
"maximum set packing" formulation is bolded: 



"The sequential technique does not guarantee the maximum 
number of nonoverlapping forms because removal of items can 
block further assembly of nonoverlapping test forms. This paper 
presents a new approach that does not have this disadvantage. The 
main idea of this approach is that if we first assemble all possible 
test forms available from the given pool (with many of the forms 
overlapping), then we can next identify the maximum subset of 
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nonoverlapping forms. The identification of the nonoverlapping 
subset can be formulated as a maximum clique problem or a 
maximum set packing problem. These problems are known as 
NP-hard (Nemhauser & Wolsey, 1988), and for a pool of usual 
size they are intractable. However, even a suboptimal solution 
(Armstrong & Belov, 2003) provides twice the number of 
nonoverlapping forms than the sequential approach. The following 
presents the new approach:" 

The problem of identifying the "maximum subset of nonoverlapping forms" can be 
formulated and solved in many ways, including the two ways mentioned in Stochastic Search, 
i.e. 1) identifying a maximum clique, and 2) identifying a maximum set packing. Ronald D. 
Armstrong did not contribute to the claimed idea of identifying the maximum subset of 
nonoverlapping forms but rather he contributed to one way of formulating the problem, i.e. the 
maximum set packing formulation; 

b) The use of comparing two different pools, on page 1 1 of the cited 
document and as set forth below in the Stochastic Search: 

"Consider two pools: Pool 1 has 1,077 passages (80 AR passages, 
85 RC passages, and 912 LR passages), and 2,418 items (747 AR 
items, 759 RC items, and 912 LR items); Pool 2 has 1,077 
passages (88 AR passages, 83 RC passages, and 906 LR passages), 
and 2,492 items (805 AR items, 781 RC items, and 906 LR items). 
Notice that the composition of both pools is similar."; 



c) Assistance in writing the summaries of the article embodied in the "Abstract," 
"Introduction" and "Conclusion", as well as provision of commentary to improve the overall 
clarity of the article. 

5. I am a co-author of a publication entitled Armstrong, R., Weissman A. and Belov, D., 
Developing and Assembling the Law School Admission Test ("Developing and Assembling the 
LSAT"). 

6. Developing and Assembling the LSAT refers to a 2002 implementation of an LSAT 
assembler utilizing an adaptive stochastic search approach. The 2002 implementation referred to 
in this article was used for internal research purposes only, at least until one year before the filing 
date of the present application, and was not used to assemble any test that was presented to, or 
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taken by, anyone outside the Law School Admission Council (LSAC), assignee of the present 
application, more than one year before the filing date of this application. The test assembler 
implemented in 2002, and its method of operation, was not disclosed to or accessible by anyone 
other than employees of or contractors to the LSAC, more than one year before the filing date of 
the present application, all of whom were under an obligation to the LSAC to maintain 
information regarding the 2002 implementation confidential. 

7. The following are contribution descriptions of publications that I co-authored: 

a) The contributions made by the authors of the presentation "A Method for 
Determining Multiple Non-Overlapping Linear Test Forms" are identified below: 

Contributions by me: The method for determining multiple non-overlapping 
linear test forms. 

Contributions by Armstrong: Implementation of the method as a sequence of 0-1 
programs and presentation of the results. 

b) The contributions made by the authors of the presentation "Combinatorial 
Analysis For Determining Item Pool Usability In Computerized Adaptive Testing" are identified 
below: 

Contributions by me: The method of the analysis and its implementation. 

Contributions by Weissman: Performing computational study and presenting the 
results. 

c) The contributions made by the authors of the presentation "A Composite Upper 
Bound for Maximum Set Packing Problem" are identified below: 

Contributions by me: The method, its implementation and presentation. 

Contributions by Weissman: Performing computational study in simulated 
Computerized Adaptive Testing environment. 

d) The contributions made by the authors of the article "Developing the Strategic 
Supply Chain for a High Stakes Testing Agency" are identified below: 

Contributions by me: The method described in section "Assembly of multiple 

forms". 
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Contributions by Armstrong: Implementing all methods described in the article 
as a sequence of 0-1 programs. 

Contributions by all authors: Sections: "Abstract", "Introduction", "An item 
response model and item life cycle", "Constraints for a single linear test form", 
"The Assembly of Multiple Overlapping Sections", "The assembly of the 
maximum number of non-overlapping sections", "The assembly of multiple non- 
overlapping linear tests", "Identifying items for development in the supply chain", 
"Discussion and conclusion", and "References". 

e) The contributions made by the authors of the article "A Method for Determining 
the Maximum Number of Nonoverlapping Linear Test Forms That Can Be Assembled from an 
Item Pool" are identified below: 

Contributions by me: The method for determining the maximum number of 
nonoverlapping linear test forms that can be assembled from an item pool 
described in section "Overview of Method". 

Contributions by Armstrong: Implementing the method as a sequence of 0-1 
programs. 

Contributions by all authors: Sections: "Executive Summary", "Abstract", 
"Introduction", "Constraints for a Single Linear Test Form", "The Assembly of 
Multiple Overlapping Sections", "The Assembly of the Maximum Number of 
Nonoverlapping Sections" , "The Assembly of Multiple Nonoverlapping Linear 
Tests", "Bounds for Large Item Pools", "Summary", and "References". 

f) The contributions made by the authors of the article "A Monte Carlo Approach to 
Design, Assemble and Evaluate Multi-Stage Adaptive Tests" are identified below: 

Contributions by me: 

Methods described in sections "Building IRT Targets", "Multi Stage Adaptive 
Test Assembly" and "Analysis of Pool and Constraints". 

Implementation of all methods presented in the article in C++. 
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Contributions by all authors: Sections "Abstract", "Introduction", "A Multiple 
Stage Adaptive Test Model", "Discussion", and "References". 

g) The contributions made by the authors of the article "A Constraint Programming 
Approach to Extract the Maximum Number of Non-Overlapping Test Forms" are identified 
below: 

Contributions by me: 

Sections "General schema for assembling maximum number of test forms", 
"Special class of MSP", "Upper bounds for MSP", "Lower bounds for MSP", 
"Filtering for MSP", "Branch-and-bound algorithm for MSP". 

Implementation of all algorithms presented in the article in C++. 

Contributions by all authors: Sections "Abstract", "Introduction", "Problem 
statement", "Computational results", "Conclusion", and "References". 

h) The contributions made by the authors of the article "Developing and Assembling 
the Law School Admission Test" are identified below: 

Contributions by me: 

Sections "LSAT Assembler", "The Test-Assembly Problem", "Monte Carlo Test 
Assembler", "Assembly of Multiple Test Forms". 

Section "Methods to Improve Random Search" in the "Appendix". 

Implementation of the above methods in C++. 

Contributions by Armstrong: Sections "The MIP Model for Assembling a Single 
LSAT" and "The Maximum Set Packing (MSP) Model" in the "Appendix". 

Contributions by Weissman: Sections "Item Response Theory" and "The Life 
Cycle of an Item". 

Contributions by all authors: Writing the following sections: "Abstract", 
"Introduction", "Model Extensions", "Computerized Adaptive Testing", 
"Conclusion", and "References". 
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i) The contributions made by the authors of the article "A Monte Carlo Approach to 
the Design, Assembly, and Evaluation of Multistage Adaptive Tests" are identified below: 

Contributions by me: Methods described in sections "Definitions", "Building 
IRT Targets", "Assembly of an MST" and "Assembly of Multiple 
Nonoverlapping MSTs", "Analysis of Pool and Constraints", "Increasing Pool 
Utilization", Appendix A, and Appendix B. 

Implementation of all methods presented in the article in C++. 

Contributions by all authors: Sections "Abstract", "Introduction", "An MST 
Model", "Discussion", and "References". 

j) The contributions made by the authors of Stochastic Search are identified above. 

k) The contributions made by the authors of the article "A Monte Carlo Approach 
for Item Pool Analysis and Design" are identified below: 

Contributions by me: The method of Monte Carlo test assembler described in 
section "Monte Carlo Test Assembler". The method of assembling multiple non- 
overlapping forms described in section "Strategies for Assembling Multiple 
Forms". The method of using frequency information generated by Monte Carlo 
test assembler: frequency of violation of selected constraints described in section 
"Constraint Difficulty and Relaxation" and item usage frequency described in 
section "Designing New Items". 

Implementation of all methods presented in the article in C++. 

Contributions by Armstrong: 

i) Use of a "maximum set packing" formulation, on page 7, in the paragraph before 
Algorithm 2. The use of "maximum set packing" formulation is bolded in the original paragraph 
from the article. 

"The sequential technique does not guarantee the maximum number of non-overlapping 
forms because removal of items can block further assembly of non-overlapping test forms. This 
paper presents a new approach that does not have this disadvantage. The main idea of this 
approach is that if we first assemble all possible test forms available from the given pool (with 
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many of the forms overlapping), then we can next identify the maximum subset of 
nonoverlapping forms. The identification of the non-overlapping subset can be formulated as a 
maximum clique problem or a maximum set packing problem. These problems are known as 
NP-hard (Nemhauser and Wolsey, 1988) and for a pool of usual size they are intractable. 
However, even a sub-optimal solution (Armstrong and Belov, 2003) provides twice the number 
of non-overlapping forms than the sequential approach. The following presents the new 
approach:" 

ii) Use of comparing two different pools, described in section "Comparison 
of Two Different Pools". 

"Consider two pools: Pool 1 has 1,077 passages (80 AR passages, 85 RC passages, and 
912 LR passages), and 2,418 items (747 AR items, 759 RC items, and 912 LR items); Pool 2 has 
1,077 passages (88 AR passages, 83 RC passages, and 906 LR passages), and 2,492 items (805 
AR items, 781 RC items, and 906 LR items). Notice that the composition of both pools is 
similar." 

Contributions by all authors: Writing of the following sections: "Abstract", 
"Introduction", "Comparison of Two Different Pools", "Conclusion", and 

"References". 

1) The contributions made by the authors of the article "Monte Carlo Test Assembly 
for Item Pool Analysis and Extension" are identified below: 

Contributions by me: The method of Monte Carlo test assembler described in 
section "Monte Carlo Test Assembler". The method of assembling multiple non- 
overlapping forms described in section "Strategies for Assembling Multiple 
Forms". The method of using frequency information generated by Monte Carlo 
test assembler: frequency of violation of selected constraints described in section 
"Constraint Difficulty and Relaxation" and item usage frequency described in 
section "Item Pool Extension". 

Implementation of all methods presented in the article in C++. 
Contributions by Armstrong: 
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i) Use of a "maximum set packing" formulation [Equation (8)], on pages 245-246, 
in the paragraph before Algorithm 2. The use of "maximum set packing" formulation is bolded 
in the original paragraph from the article. 

"The sequential technique does not guarantee the maximum number of non-overlapping 
tests because removal of items from the pool can block further assembly of non-overlapping 
tests. This paper presents a new approach that does not have this disadvantage. The main idea 
of this approach is that if we first assemble a large number of tests from the given pool (with 
many of the tests overlapping), then we can next identify the maximum subset of non- 
overlapping tests. The identification of such subset for given tests (sections) can be 
formulated as the following 0-1 program: 

^ y } —> max 

7=1 

£a,v ; <l i = l,...,m> (1) 
v,e{0,l} y = l #i 

where n is number of overlapping tests (sections), m is number of passages, variable 
yj = 1 when the /* test (section) is included in solution and y. = 0, otherwise, a y = 1 if the 
j th test (section) contains the i"' passage and a tj = 0 , otherwise. Problem (1) is a maximum 
set packing problem, known to be NP-hard (Nemhauser & Wolsey, 1988) and for a pool of 
usual size it is intractable. However, even a sub-optimal solution (Armstrong & Belov, 2003) 
provides almost twice the number of non-overlapping tests as the sequential approach. The 
following presents the new approach:" 

ii) Use of comparing two different pools, described in section "Comparison 
of Two Different Pools". 

"Consider two pools: Pool 1 has 1,077 passages (80 AR passages, 85 RC passages, 912 
LR passages) and 2,418 items (747 AR items, 759 RC items, 912 LR items); Pool 2 has 1,077 
passages (88 AR passages, 83 RC passages, 906 LR passages) and 2,492 items (805 AR items, 
781 RC items, 906 LR items). Notice that the composition of both pools is similar." 
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Contributions by all authors: Sections: "Abstract", "Introduction", "Comparison 
of Two Different Pools", "Conclusion", and "References". 

m) The contributions made by the authors of the article "A Monte Carlo Approach 
for Adaptive Testing with Content Constraints" are identified below: 

Contributions by me: Methods described in sections "Monte Carlo CAT" and the 
"Appendix". 

Implementation of all methods presented in the article in C++. 

Contributions by all authors: Sections "Abstract", "Introduction", "Computer 
Experiments", "Summary", and "References". 

n) The contributions made by the authors of the article "Direct and Inverse Problems 
of Item Pool Design for Computerized Adaptive Testing" are identified below: 

Contributions by me: Methods described in sections "Uniform Test Assembly", 
"Information Threshold", "Definitions", "Direct Problem and Direct Algorithm", 
and "Inverse Problem and Inverse Algorithm". 

Implementation of all methods presented in the article in C++. 

Contributions by all authors: Sections "Abstract", "Introduction", "Test 
Assembly Constraints", "Computer Experiments", "Discussion", and 
"References". 

o) The article "The Assembly and Administration of the Law School Admission 
Test" is a non-published draft of the article referenced in subsection h) above. After the first 
revision of the article, the title of the article was changed to "Developing and Assembling the 
Law School Admission Test". The same contributions listed above in subsection h) for 
"Developing and Assembling the Law School Admission Test" apply to "The Assembly and 
Administration of the Law School Admission Test". 

8. I relied on two publications (a and b below) to develop the subject matter claimed in the 
present application. These publications describe the following techniques: "divide-and- 
conquer", "local search", and "tabu search". 
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a) Aho, A. V., Hopcroft, J. E., & Ullman, J. D. (1983). Data structures and algorithms. 
Reading, MA: Addison- Wesley. 

The following claims rely upon the "divide-and-conquer" technique, described on 
pages 306-307: 3, 4, 5, 6, 7, 10; 15, 16, 17, 18, 19, 22. 

The following claims rely upon the "local search" technique, described on pages 
336-337: 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12; 13, 14, 15, 16, 17, 18, 19, 20, 21, 23, 24. 

b) Glover, F., Taillard, E., & de Werra, D. (1993). A user's guide to tabu search. Annals 
of Operations Research, 41, 1-28. 



9. Developing and Assembling the LSAT describes the Monte Carlo test assembler which, 
as described herein in paragraph 6, was used by LSAC for research purposes since 2002. Use of 
the Monte Carlo methods for test assembly is part of the claimed invention. The Monte Carlo 
test assembler was used at LSAC for research purposes beginning in 2002. In August of 2003, 
the Monte Carlo test assembler was used by LSAC to produce LSAT forms that were 
administered to actual test-takers. There are no other LSAC products or services that incorporate 
the claimed subject matter. 

I hereby declare that all statements made herein of my own knowledge are true and that 
all statements made on information and belief are believed to be true; and further that these 
statements are punishable by fine or imprisonment, or both, under Section 1001 of Title 1 8 of the 
United States Code, and that such willful statements may jeopardize the validity of the 
application or any patent issued thereon. 



The following claims rely upon the "tabu search" technique: 8, 9, 20 and 21. 





Dmitry I. Belov 
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